Skip to content

MGMT-21200: Enable dual-stack#390

Merged
openshift-merge-bot[bot] merged 1 commit intorh-ecosystem-edge:mainfrom
danmanor:enable-dual-stack
Jul 31, 2025
Merged

MGMT-21200: Enable dual-stack#390
openshift-merge-bot[bot] merged 1 commit intorh-ecosystem-edge:mainfrom
danmanor:enable-dual-stack

Conversation

@danmanor
Copy link
Copy Markdown
Member

@danmanor danmanor commented Jul 22, 2025

Dual Stack Networking Support for Recert

This PR implements comprehensive dual stack networking support in the recert tool, enabling transformation from one dual stack configuration to another with conditional replacement logic.

Overview

The recert tool now supports dual stack networking configurations, allowing clusters to operate with both IPv4 and IPv6 addresses simultaneously. This includes intelligent conditional replacement that only modifies resources when the original IP addresses are actually present.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Conditional Replacement Logic (Major Enhancement)

Problem Solved

Previously, etcd resource modification functions would unconditionally replace IP addresses, potentially corrupting configurations when original IPs were not present.

Solution Implemented

  • Conditional replacement: All etcd modification functions now check if the original IP exists before performing replacements
  • Fail-safe behavior: Functions skip replacement and log informational messages when original IPs are not found
  • No failures on mismatches: Missing original IPs no longer cause the process to fail

Functions Updated with Conditional Logic

  • fix_etcd_endpoints: Only replaces endpoint IPs when original IP matches current value
  • fix_openshift_apiserver_configmap: Checks storage config URLs before modification
  • fix_openshiftapiservers_cluster: Validates storage config contains original IP
  • fix_networks_cluster: Verifies annotation values before updating
  • fix_etcd_member: Fetches actual member configuration and validates peer URLs
  • fix_storage_config: Helper function now requires original IP verification

Technical Implementation

// Example: Conditional replacement pattern
if current_value == original_ip {
    // Perform replacement
    data[key] = new_ip;
    put_etcd_yaml(etcd_client, &location, resource).await?;
} else {
    log::info!("Original IP {} not found, current value is {}, skipping replacement", 
               original_ip, current_value);
}

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Dual Stack IPs:
    • First IP (IPv4) processed through all etcd resources
    • Second IP (IPv6) processed through all etcd resources
    • Each IP change is logged for traceability

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Etcd Resource Management

  1. Original IP extraction: Retrieves current IPs from node configuration before making changes
  2. Sequential processing: IPv4 and IPv6 changes applied in separate passes
  3. Resource isolation: Each IP family processed independently to prevent cross-contamination

Function Reorganization

  • Centralized etcd modifications: All etcd resource changes moved to fix_etcd_resources_for_ip_pair
  • Consistent API: All functions now take both original_ip and new_ip parameters
  • Improved maintainability: Common patterns extracted and standardized

Backward Compatibility

  • Single IP support: Existing single IP configurations continue to work unchanged
  • CLI compatibility: Single IP values work as before, arrays enable dual stack
  • Configuration files: Both single values and arrays supported in YAML

Benefits

  1. Robust dual stack support: Full IPv4/IPv6 dual stack cluster transformations
  2. Safe operations: Conditional replacement prevents accidental corruption
  3. Better observability: Detailed logging shows what changes are made or skipped
  4. Maintainable code: Consistent patterns across all etcd modification functions
  5. Future-proof: Architecture supports easy addition of new dual stack features

Testing

  • All existing single IP tests continue to pass
  • New dual stack scenarios covered by expanded test cases
  • Conditional replacement logic prevents regressions from missing original IPs

Summary by CodeRabbit

  • New Features

    • Added support for dual-stack (IPv4 and IPv6) network configurations, including handling multiple IP addresses and network CIDRs throughout the configuration and processing logic.
    • Enhanced IP renaming and network CIDR update processes to work with both IPv4 and IPv6 addresses for improved cluster compatibility.
  • Bug Fixes

    • Improved robustness of IP and CIDR replacement logic to ensure changes only occur when the original values are present.
  • Documentation

    • Updated configuration documentation to clarify dual-stack usage and ordering requirements for IP addresses and CIDRs.
  • Style

    • Standardized error handling for better consistency and maintainability.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This document describes the implementation of dual stack networking support in the recert tool.

Overview

The recert tool now supports dual stack networking, allowing transformation from one dual stack configuration to another. Dual stack means that the cluster can use both IPv4 and IPv6 addresses simultaneously.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Configuration Examples

YAML Configuration (dual stack)

ip:
 - 192.168.126.99    # IPv4 first
 - 2001:db8::99      # IPv6 second

machine_network_cidr:
 - 192.168.126.0/24  # IPv4 machine network
 - 2001:db8::/64     # IPv6 machine network

install_config: |
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23
   - cidr: fd01::/48      # IPv6 cluster network
     hostPrefix: 64
   machineNetwork:
   - cidr: 192.168.126.0/24
   - cidr: 2001:db8::/64   # IPv6 machine network
   serviceNetwork:
   - 172.30.0.0/16
   - fd02::/112           # IPv6 service network

CLI Usage (dual stack)

./recert --ip 192.168.126.99,2001:db8::99 \
        --machine-network-cidr 192.168.126.0/24,2001:db8::/64 \
        --other-options...

YAML Configuration (single stack - backward compatible)

ip: 192.168.126.99
machine_network_cidr: 192.168.126.0/24
# OR
ip:
 - 192.168.126.99
machine_network_cidr:
 - 192.168.126.0/24

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Multiple IPs: Each IP is processed in sequence, with logging for each step
  3. Order matters: For dual stack, IPv4 should be specified first, then IPv6

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Usage Examples

Converting Single Stack to Dual Stack

# Original cluster has IPv4 only
./recert --ip 192.168.1.100,2001:db8::100 \
        --machine-network-cidr 192.168.1.0/24,2001:db8::/64 \
        --other-config...

Converting Dual Stack to Dual Stack

# Transform from one dual stack config to another
./recert --ip 10.0.1.100,fd00::100 \
        --machine-network-cidr 10.0.1.0/24,fd00::/64 \
        --other-config...

Using Configuration File

# recert-config.yaml
ip:
 - 10.0.1.100
 - fd00::100
machine_network_cidr:
 - 10.0.1.0/24
 - fd00::/64
install_config: |
 # Complete install config with dual stack networking
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23  
   - cidr: fd01::/48
     hostPrefix: 64
   # ... rest of config

Benefits

  1. Full dual stack support: Handle clusters with both IPv4 and IPv6 connectivity
  2. Backward compatibility: Single IP configurations continue to work
  3. Flexible input: Support both CLI and YAML configuration methods
  4. OpenShift compliance: Follows OpenShift dual stack networking patterns

Important Notes

  • IP order matters: Always specify IPv4 first, then IPv6 for dual stack
  • Network alignment: Ensure VIP addresses are within the corresponding machine networks
  • Validation: The tool will process each IP separately, so both must be valid
  • Logging: Enhanced logging shows progress through each IP transformation

Testing

The implementation maintains full backward compatibility while adding dual stack support. Test scenarios should include:

  1. Single IPv4 -> Single IPv4 (existing functionality)
  2. Single IPv6 -> Single IPv6 (existing functionality)
  3. Dual stack -> Dual stack (new functionality)
  4. Mixed configurations in install-config

This implementation enables the recert tool to handle modern OpenShift dual stack networking requirements while maintaining compatibility with existing single stack deployments.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from omertuc and tsorya July 22, 2025 07:31
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Jul 22, 2025

Walkthrough

This change introduces dual-stack (IPv4 and IPv6) support for cluster IP and machine network CIDR configuration throughout the codebase and configuration files. It updates struct fields, CLI parameters, and YAML configuration to accept lists of addresses/CIDRs. The etcd and postprocessing logic is refactored to handle dual-stack renaming and resource updates, and robust checks are added for IP replacements.

Changes

Cohort / File(s) Change Summary
YAML Dual-Stack Config
hack/dummy_config.yaml
Updated configuration to support dual-stack IPs and CIDRs by converting single-value fields to lists for IPs and network CIDRs, and adding IPv6 entries throughout.
Config Structs and Parsing
src/config.rs, src/config/cli.rs
Renamed and changed types of ip/machine_network_cidr fields to vector form in structs and CLI, updated parsing logic to accept lists or single values, and updated documentation.
etcd Encoding Support
src/etcd_encoding.rs
Added support for Kubernetes Node resource encoding/decoding by introducing a new wrapper struct and extending the decode logic.
Cluster Customization & Postprocessing
src/ocp_postprocess.rs
Refactored to handle lists of IPs/CIDRs, added dual-stack renaming logic, and updated logging and control flow for dual-stack support.
IP Renaming Logic (Dual-Stack)
src/ocp_postprocess/ip_rename.rs
Added dual-stack renaming functions for etcd and filesystem resources, refactored existing logic, and introduced helper functions for extracting and replacing both IPv4 and IPv6 addresses.
etcd Rename Utilities (Dual-Stack)
src/ocp_postprocess/ip_rename/etcd_rename.rs
Added dual-stack IP extraction, updated all resource fix functions to handle both original and new IPs, improved IPv6 formatting, and added robust checks for replacements.
Machine Network CIDR Rename
src/ocp_postprocess/machine_config_cidr_rename/etcd_rename.rs
Removed restriction on single CIDR, added logic to update machine network with multiple CIDRs for dual-stack support.
Error Handling Consistency
src/ocp_postprocess/rename_utils.rs
Replaced explicit error checks with the ensure! macro for more consistent error assertion in several utility functions.

Sequence Diagram(s)

sequenceDiagram
    participant CLI
    participant ConfigParser
    participant PostProcess
    participant EtcdClient
    participant FileSystem

    CLI->>ConfigParser: Parse config (IPs, CIDRs as lists)
    ConfigParser->>PostProcess: Provide parsed IPs/CIDRs
    PostProcess->>EtcdClient: Extract original IPs (dual-stack)
    alt Single IP
        PostProcess->>EtcdClient: Rename resources (IPv4)
        PostProcess->>FileSystem: Rename files/dirs (IPv4)
    else Dual-stack IPs
        PostProcess->>EtcdClient: Rename resources (IPv4 & IPv6)
        PostProcess->>FileSystem: Rename files/dirs (IPv4 & IPv6)
    end
    PostProcess->>EtcdClient: Update machine network CIDRs
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~40 minutes

Suggested labels

lgtm

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 368516c and 3ee6a5a.

📒 Files selected for processing (9)
  • hack/dummy_config.yaml (1 hunks)
  • src/config.rs (6 hunks)
  • src/config/cli.rs (2 hunks)
  • src/etcd_encoding.rs (3 hunks)
  • src/ocp_postprocess.rs (3 hunks)
  • src/ocp_postprocess/ip_rename.rs (3 hunks)
  • src/ocp_postprocess/ip_rename/etcd_rename.rs (9 hunks)
  • src/ocp_postprocess/machine_config_cidr_rename/etcd_rename.rs (2 hunks)
  • src/ocp_postprocess/rename_utils.rs (5 hunks)
🚧 Files skipped from review as they are similar to previous changes (7)
  • src/ocp_postprocess/rename_utils.rs
  • src/ocp_postprocess/machine_config_cidr_rename/etcd_rename.rs
  • src/etcd_encoding.rs
  • hack/dummy_config.yaml
  • src/config/cli.rs
  • src/ocp_postprocess.rs
  • src/config.rs
🧰 Additional context used
🧬 Code Graph Analysis (1)
src/ocp_postprocess/ip_rename/etcd_rename.rs (2)
src/k8s_etcd.rs (4)
  • new (34-47)
  • put_etcd_yaml (329-334)
  • get_etcd_json (316-327)
  • key (303-303)
src/ocp_postprocess/cluster_domain_rename/etcd_rename.rs (3)
  • fix_openshift_apiserver_configmap (332-357)
  • config (1083-1085)
  • data (1072-1079)
🔇 Additional comments (14)
src/ocp_postprocess/ip_rename.rs (5)

2-2: LGTM! Good addition of ensure! macro import.

The import of ensure! macro from anyhow is appropriate and aligns with the feedback from past reviews to use ensure! instead of if ... { bail!(...) } patterns.


93-96: LGTM! Consistent with single-stack implementation.

The dual-stack version maintains consistency with the single-stack fix_file_resources function by being a no-op, which is appropriate.


98-148: LGTM! Good refactoring to centralize etcd modifications.

This new function properly centralizes all etcd resource modification calls with a consistent API that accepts both original and new IP parameters. This improves maintainability and supports the conditional replacement logic.


150-167: LGTM! Proper single-stack validation and refactoring.

The refactored function correctly:

  • Extracts original IPs using the shared function
  • Validates exactly 1 IP for single-stack using ensure!
  • Delegates to the centralized fix_etcd_resources_for_ip_pair function

169-181: LGTM! Proper dual-stack validation.

The function correctly validates that exactly 2 IPs are present for dual-stack processing using the ensure! macro.

src/ocp_postprocess/ip_rename/etcd_rename.rs (9)

9-17: LGTM! Clean IPv6 detection utility function.

The is_ipv6 function properly uses the standard library's IP address parsing and returns appropriate error context. The implementation is robust and handles edge cases well.


19-83: LGTM! Comprehensive dual-stack IP extraction logic.

The implementation correctly:

  • Validates exactly one node in the cluster
  • Extracts both IPv4 and IPv6 InternalIP addresses from node configuration
  • Maintains order (IPv4 first, IPv6 second) as documented in PR objectives
  • Provides informative logging for both single-stack and dual-stack scenarios
  • Uses proper error handling and validation

85-133: LGTM! Excellent conditional replacement logic.

The enhanced function now:

  • Accepts both original and new IP parameters for better API consistency
  • Implements proper conditional checks to prevent configuration corruption
  • Handles IPv6 bracket formatting correctly
  • Provides informative logging when original IP is not found
  • Only modifies etcd when the original IP is actually present

This addresses the key PR objective of safe operations through conditional replacements.


135-164: LGTM! Robust storage config replacement with proper validation.

The function implements excellent conditional replacement logic:

  • Properly formats IPv6 addresses with brackets
  • Validates the original IP exists before making changes
  • Uses iterator methods to safely check URL arrays
  • Provides clear logging when replacements are skipped

216-271: LGTM! Safe etcd endpoints replacement with validation.

The updated function properly:

  • Accepts original and new IP parameters
  • Only replaces values when the original IP is found
  • Provides informative logging when skipping replacements
  • Maintains the existing concurrent processing pattern

443-456: LGTM! Consistent API update.

The function signature update to accept both original and new IP parameters is consistent with the broader refactoring and delegates properly to the updated fix_storage_config function.


475-537: LGTM! Excellent conditional replacement for OAuth API server.

The enhanced logic properly:

  • Handles both IPv4 and IPv6 etcd server arguments with correct formatting
  • Uses conditional replacement only when the original IP is found
  • Provides informative logging when arguments are not found (expected for IPv6 in dual-stack)
  • Maintains proper error handling throughout

539-571: LGTM! Safe annotation replacement with proper validation.

The function implements robust conditional replacement:

  • Checks for annotation existence before attempting replacement
  • Only replaces when the original IP matches exactly
  • Provides clear logging for both missing annotations and value mismatches
  • Handles the optional nature of the annotation gracefully

573-620: LGTM! Dynamic etcd member validation and update.

The new implementation is much more robust:

  • Queries the actual etcd cluster member list for validation
  • Properly handles IPv6 URL formatting with brackets
  • Only updates when the original IP is found in peer URLs
  • Uses the existing update_member method appropriately
  • Provides detailed logging for troubleshooting

This is a significant improvement over static replacement approaches.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Jul 22, 2025

Hi @danmanor. Thanks for your PR.

I'm waiting for a rh-ecosystem-edge member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This is a description of the implementation of dual stack networking support in the recert tool.

Overview

The recert tool now supports dual stack networking, allowing transformation from one dual stack configuration to another. Dual stack means that the cluster can use both IPv4 and IPv6 addresses simultaneously.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Configuration Examples

YAML Configuration (dual stack)

ip:
 - 192.168.126.99    # IPv4 first
 - 2001:db8::99      # IPv6 second

machine_network_cidr:
 - 192.168.126.0/24  # IPv4 machine network
 - 2001:db8::/64     # IPv6 machine network

install_config: |
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23
   - cidr: fd01::/48      # IPv6 cluster network
     hostPrefix: 64
   machineNetwork:
   - cidr: 192.168.126.0/24
   - cidr: 2001:db8::/64   # IPv6 machine network
   serviceNetwork:
   - 172.30.0.0/16
   - fd02::/112           # IPv6 service network

CLI Usage (dual stack)

./recert --ip 192.168.126.99,2001:db8::99 \
        --machine-network-cidr 192.168.126.0/24,2001:db8::/64 \
        --other-options...

YAML Configuration (single stack - backward compatible)

ip: 192.168.126.99
machine_network_cidr: 192.168.126.0/24
# OR
ip:
 - 192.168.126.99
machine_network_cidr:
 - 192.168.126.0/24

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Multiple IPs: Each IP is processed in sequence, with logging for each step
  3. Order matters: For dual stack, IPv4 should be specified first, then IPv6

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Usage Examples

Converting Single Stack to Dual Stack

# Original cluster has IPv4 only
./recert --ip 192.168.1.100,2001:db8::100 \
        --machine-network-cidr 192.168.1.0/24,2001:db8::/64 \
        --other-config...

Converting Dual Stack to Dual Stack

# Transform from one dual stack config to another
./recert --ip 10.0.1.100,fd00::100 \
        --machine-network-cidr 10.0.1.0/24,fd00::/64 \
        --other-config...

Using Configuration File

# recert-config.yaml
ip:
 - 10.0.1.100
 - fd00::100
machine_network_cidr:
 - 10.0.1.0/24
 - fd00::/64
install_config: |
 # Complete install config with dual stack networking
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23  
   - cidr: fd01::/48
     hostPrefix: 64
   # ... rest of config

Benefits

  1. Full dual stack support: Handle clusters with both IPv4 and IPv6 connectivity
  2. Backward compatibility: Single IP configurations continue to work
  3. Flexible input: Support both CLI and YAML configuration methods
  4. OpenShift compliance: Follows OpenShift dual stack networking patterns

Important Notes

  • IP order matters: Always specify IPv4 first, then IPv6 for dual stack
  • Network alignment: Ensure VIP addresses are within the corresponding machine networks
  • Validation: The tool will process each IP separately, so both must be valid
  • Logging: Enhanced logging shows progress through each IP transformation

Testing

The implementation maintains full backward compatibility while adding dual stack support. Test scenarios should include:

  1. Single IPv4 -> Single IPv4 (existing functionality)
  2. Single IPv6 -> Single IPv6 (existing functionality)
  3. Dual stack -> Dual stack (new functionality)
  4. Mixed configurations in install-config

This implementation enables the recert tool to handle modern OpenShift dual stack networking requirements while maintaining compatibility with existing single stack deployments.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This is a description of the implementation of dual stack networking support in the recert tool.

Overview

The recert tool now supports dual stack networking, allowing transformation from one dual stack configuration to another. Dual stack means that the cluster can use both IPv4 and IPv6 addresses simultaneously.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Configuration Examples

YAML Configuration (dual stack)

ip:
 - 192.168.126.99    # IPv4 first
 - 2001:db8::99      # IPv6 second

machine_network_cidr:
 - 192.168.126.0/24  # IPv4 machine network
 - 2001:db8::/64     # IPv6 machine network

install_config: |
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23
   - cidr: fd01::/48      # IPv6 cluster network
     hostPrefix: 64
   machineNetwork:
   - cidr: 192.168.126.0/24
   - cidr: 2001:db8::/64   # IPv6 machine network
   serviceNetwork:
   - 172.30.0.0/16
   - fd02::/112           # IPv6 service network

CLI Usage (dual stack)

./recert --ip 192.168.126.99,2001:db8::99 \
        --machine-network-cidr 192.168.126.0/24,2001:db8::/64 \
        --other-options...

YAML Configuration (single stack - backward compatible)

ip: 192.168.126.99
machine_network_cidr: 192.168.126.0/24
# OR
ip:
 - 192.168.126.99
machine_network_cidr:
 - 192.168.126.0/24

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Multiple IPs: Each IP is processed in sequence, with logging for each step
  3. Order matters: For dual stack, IPv4 should be specified first, then IPv6

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Usage Examples

Converting Single Stack to Dual Stack

# Original cluster has IPv4 only
./recert --ip 192.168.1.100,2001:db8::100 \
        --machine-network-cidr 192.168.1.0/24,2001:db8::/64 \
        --other-config...

Converting Dual Stack to Dual Stack

# Transform from one dual stack config to another
./recert --ip 10.0.1.100,fd00::100 \
        --machine-network-cidr 10.0.1.0/24,fd00::/64 \
        --other-config...

Using Configuration File

# recert-config.yaml
ip:
 - 10.0.1.100
 - fd00::100
machine_network_cidr:
 - 10.0.1.0/24
 - fd00::/64
install_config: |
 # Complete install config with dual stack networking
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23  
   - cidr: fd01::/48
     hostPrefix: 64
   # ... rest of config

Benefits

  1. Full dual stack support: Handle clusters with both IPv4 and IPv6 connectivity
  2. Backward compatibility: Single IP configurations continue to work
  3. Flexible input: Support both CLI and YAML configuration methods
  4. OpenShift compliance: Follows OpenShift dual stack networking patterns

Important Notes

  • IP order matters: Always specify IPv4 first, then IPv6 for dual stack
  • Network alignment: Ensure VIP addresses are within the corresponding machine networks
  • Validation: The tool will process each IP separately, so both must be valid
  • Logging: Enhanced logging shows progress through each IP transformation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This is a description of the implementation of dual stack networking support in the recert tool.

Overview

The recert tool now supports dual stack networking, allowing transformation from one dual stack configuration to another. Dual stack means that the cluster can use both IPv4 and IPv6 addresses simultaneously.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Configuration Examples

YAML Configuration (dual stack)

ip:
 - 192.168.126.99    # IPv4 first
 - 2001:db8::99      # IPv6 second

machine_network_cidr:
 - 192.168.126.0/24  # IPv4 machine network
 - 2001:db8::/64     # IPv6 machine network

install_config: |
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23
   - cidr: fd01::/48      # IPv6 cluster network
     hostPrefix: 64
   machineNetwork:
   - cidr: 192.168.126.0/24
   - cidr: 2001:db8::/64   # IPv6 machine network
   serviceNetwork:
   - 172.30.0.0/16
   - fd02::/112           # IPv6 service network

CLI Usage (dual stack)

./recert --ip 192.168.126.99,2001:db8::99 \
        --machine-network-cidr 192.168.126.0/24,2001:db8::/64 \
        --other-options...

YAML Configuration (single stack - backward compatible)

ip: 192.168.126.99
machine_network_cidr: 192.168.126.0/24
# OR
ip:
 - 192.168.126.99
machine_network_cidr:
 - 192.168.126.0/24

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Multiple IPs: Each IP is processed in sequence, with logging for each step
  3. Order matters: For dual stack, IPv4 should be specified first, then IPv6

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Usage Examples

Converting Single Stack to Dual Stack

# Original cluster has IPv4 only
./recert --ip 192.168.1.100,2001:db8::100 \
        --machine-network-cidr 192.168.1.0/24,2001:db8::/64 \
        --other-config...

Converting Dual Stack to Dual Stack

# Transform from one dual stack config to another
./recert --ip 10.0.1.100,fd00::100 \
        --machine-network-cidr 10.0.1.0/24,fd00::/64 \
        --other-config...

Using Configuration File

# recert-config.yaml
ip:
 - 10.0.1.100
 - fd00::100
machine_network_cidr:
 - 10.0.1.0/24
 - fd00::/64
install_config: |
 # Complete install config with dual stack networking
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23  
   - cidr: fd01::/48
     hostPrefix: 64
   # ... rest of config

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This is a description of the implementation of dual stack networking support in the recert tool.

Overview

The recert tool now supports dual stack networking, allowing transformation from one dual stack configuration to another. Dual stack means that the cluster can use both IPv4 and IPv6 addresses simultaneously.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Configuration Examples

YAML Configuration (dual stack)

ip:
 - 192.168.126.99    # IPv4 first
 - 2001:db8::99      # IPv6 second

machine_network_cidr:
 - 192.168.126.0/24  # IPv4 machine network
 - 2001:db8::/64     # IPv6 machine network

install_config: |
 networking:
   clusterNetwork:
   - cidr: 10.128.0.0/14
     hostPrefix: 23
   - cidr: fd01::/48      # IPv6 cluster network
     hostPrefix: 64
   machineNetwork:
   - cidr: 192.168.126.0/24
   - cidr: 2001:db8::/64   # IPv6 machine network
   serviceNetwork:
   - 172.30.0.0/16
   - fd02::/112           # IPv6 service network

CLI Usage (dual stack)

./recert --ip 192.168.126.99,2001:db8::99 \
        --machine-network-cidr 192.168.126.0/24,2001:db8::/64 \
        --other-options...

YAML Configuration (single stack - backward compatible)

ip: 192.168.126.99
machine_network_cidr: 192.168.126.0/24
# OR
ip:
 - 192.168.126.99
machine_network_cidr:
 - 192.168.126.0/24

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Multiple IPs: Each IP is processed in sequence, with logging for each step
  3. Order matters: For dual stack, IPv4 should be specified first, then IPv6

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This is a description of the implementation of dual stack networking support in the recert tool.

Overview

The recert tool now supports dual stack networking, allowing transformation from one dual stack configuration to another. Dual stack means that the cluster can use both IPv4 and IPv6 addresses simultaneously.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Multiple IPs: Each IP is processed in sequence, with logging for each step
  3. Order matters: For dual stack, IPv4 should be specified first, then IPv6

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This is a description of the implementation of dual stack networking support in the recert tool.

Overview

The recert tool now supports dual stack networking, allowing transformation from one dual stack configuration to another. Dual stack means that the cluster can use both IPv4 and IPv6 addresses simultaneously.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Multiple IPs: Each IP is processed in sequence, with logging for each step

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This is a description of the implementation of dual stack networking support in the recert tool.

Overview

The recert tool now supports dual stack networking, allowing transformation from one dual stack configuration to another.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Multiple IPs: Each IP is processed in sequence, with logging for each step

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@danmanor danmanor force-pushed the enable-dual-stack branch from fd0fe1d to c4a40a4 Compare July 22, 2025 07:35
@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This PR implements comprehensive dual stack networking support in the recert tool, enabling transformation from one dual stack configuration to another with conditional replacement logic.

Overview

The recert tool now supports dual stack networking configurations, allowing clusters to operate with both IPv4 and IPv6 addresses simultaneously. This includes intelligent conditional replacement that only modifies resources when the original IP addresses are actually present.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Conditional Replacement Logic (Major Enhancement)

Problem Solved

Previously, etcd resource modification functions would unconditionally replace IP addresses, potentially corrupting configurations when original IPs were not present.

Solution Implemented

  • Conditional replacement: All etcd modification functions now check if the original IP exists before performing replacements
  • Fail-safe behavior: Functions skip replacement and log informational messages when original IPs are not found
  • No failures on mismatches: Missing original IPs no longer cause the process to fail

Functions Updated with Conditional Logic

  • fix_etcd_endpoints: Only replaces endpoint IPs when original IP matches current value
  • fix_openshift_apiserver_configmap: Checks storage config URLs before modification
  • fix_openshiftapiservers_cluster: Validates storage config contains original IP
  • fix_networks_cluster: Verifies annotation values before updating
  • fix_etcd_member: Fetches actual member configuration and validates peer URLs
  • fix_storage_config: Helper function now requires original IP verification

Technical Implementation

// Example: Conditional replacement pattern
if current_value == original_ip {
   // Perform replacement
   data[key] = new_ip;
   put_etcd_yaml(etcd_client, &location, resource).await?;
} else {
   log::info!("Original IP {} not found, current value is {}, skipping replacement", 
              original_ip, current_value);
}

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Dual Stack IPs:
  • First IP (IPv4) processed through all etcd resources
  • Second IP (IPv6) processed through all etcd resources
  • Each IP change is logged for traceability

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Etcd Resource Management

  1. Original IP extraction: Retrieves current IPs from node configuration before making changes
  2. Sequential processing: IPv4 and IPv6 changes applied in separate passes
  3. Resource isolation: Each IP family processed independently to prevent cross-contamination

Function Reorganization

  • Centralized etcd modifications: All etcd resource changes moved to fix_etcd_resources_for_ip_pair
  • Consistent API: All functions now take both original_ip and new_ip parameters
  • Improved maintainability: Common patterns extracted and standardized

Backward Compatibility

  • Single IP support: Existing single IP configurations continue to work unchanged
  • CLI compatibility: Single IP values work as before, arrays enable dual stack
  • Configuration files: Both single values and arrays supported in YAML

Benefits

  1. Robust dual stack support: Full IPv4/IPv6 dual stack cluster transformations
  2. Safe operations: Conditional replacement prevents accidental corruption
  3. Better observability: Detailed logging shows what changes are made or skipped
  4. Maintainable code: Consistent patterns across all etcd modification functions
  5. Future-proof: Architecture supports easy addition of new dual stack features

Testing

  • All existing single IP tests continue to pass
  • New dual stack scenarios covered by expanded test cases
  • Conditional replacement logic prevents regressions from missing original IPs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This PR implements comprehensive dual stack networking support in the recert tool, enabling transformation from one dual stack configuration to another with conditional replacement logic.

Overview

The recert tool now supports dual stack networking configurations, allowing clusters to operate with both IPv4 and IPv6 addresses simultaneously. This includes intelligent conditional replacement that only modifies resources when the original IP addresses are actually present.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Conditional Replacement Logic (Major Enhancement)

Problem Solved

Previously, etcd resource modification functions would unconditionally replace IP addresses, potentially corrupting configurations when original IPs were not present.

Solution Implemented

  • Conditional replacement: All etcd modification functions now check if the original IP exists before performing replacements
  • Fail-safe behavior: Functions skip replacement and log informational messages when original IPs are not found
  • No failures on mismatches: Missing original IPs no longer cause the process to fail

Functions Updated with Conditional Logic

  • fix_etcd_endpoints: Only replaces endpoint IPs when original IP matches current value
  • fix_openshift_apiserver_configmap: Checks storage config URLs before modification
  • fix_openshiftapiservers_cluster: Validates storage config contains original IP
  • fix_networks_cluster: Verifies annotation values before updating
  • fix_etcd_member: Fetches actual member configuration and validates peer URLs
  • fix_storage_config: Helper function now requires original IP verification

Technical Implementation

// Example: Conditional replacement pattern
if current_value == original_ip {
   // Perform replacement
   data[key] = new_ip;
   put_etcd_yaml(etcd_client, &location, resource).await?;
} else {
   log::info!("Original IP {} not found, current value is {}, skipping replacement", 
              original_ip, current_value);
}

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Dual Stack IPs:
  • First IP (IPv4) processed through all etcd resources
  • Second IP (IPv6) processed through all etcd resources
  • Each IP change is logged for traceability

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Etcd Resource Management

  1. Original IP extraction: Retrieves current IPs from node configuration before making changes
  2. Sequential processing: IPv4 and IPv6 changes applied in separate passes
  3. Resource isolation: Each IP family processed independently to prevent cross-contamination

Function Reorganization

  • Centralized etcd modifications: All etcd resource changes moved to fix_etcd_resources_for_ip_pair
  • Consistent API: All functions now take both original_ip and new_ip parameters
  • Improved maintainability: Common patterns extracted and standardized

Backward Compatibility

  • Single IP support: Existing single IP configurations continue to work unchanged
  • CLI compatibility: Single IP values work as before, arrays enable dual stack
  • Configuration files: Both single values and arrays supported in YAML

Benefits

  1. Robust dual stack support: Full IPv4/IPv6 dual stack cluster transformations
  2. Safe operations: Conditional replacement prevents accidental corruption
  3. Better observability: Detailed logging shows what changes are made or skipped
  4. Maintainable code: Consistent patterns across all etcd modification functions
  5. Future-proof: Architecture supports easy addition of new dual stack features

Testing

  • All existing single IP tests continue to pass
  • New dual stack scenarios covered by expanded test cases
  • Conditional replacement logic prevents regressions from missing original IPs

Summary by CodeRabbit

  • New Features

  • Added support for dual-stack (IPv4 and IPv6) cluster configuration, including handling multiple IP addresses and network CIDRs in both configuration files and command-line inputs.

  • Introduced comprehensive dual-stack IP renaming functionality, ensuring both IPv4 and IPv6 addresses are processed in etcd and filesystem resources.

  • Enhanced etcd encoding/decoding to support additional Kubernetes resource types.

  • Documentation

  • Updated command-line option descriptions to clarify input formats for dual-stack IPs and CIDRs.

  • Bug Fixes

  • Improved logic to ensure IP and CIDR replacements only occur when the original values are present, with robust handling for both IPv4 and IPv6 scenarios.

  • Refactor

  • Streamlined internal logic for processing multiple network entries and improved logging for dual-stack operations.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (4)
src/ocp_postprocess/machine_config_cidr_rename/etcd_rename.rs (1)

34-42: Consider adding CIDR validation.

The dual-stack CIDR handling logic is correct and properly handles comma-separated inputs. However, consider adding validation to ensure each CIDR is valid and not empty after trimming.

    let new_networks: Vec<_> = machine_config_network
        .split(',')
+       .filter(|cidr| !cidr.trim().is_empty())
        .map(|cidr| serde_json::json!({"cidr": cidr.trim()}))
        .collect();
src/config.rs (2)

225-238: Consider refactoring duplicated parsing logic.

The parsing logic for handling both single strings and arrays is duplicated between the ip and machine_network_cidr fields. Consider extracting this into a helper function.

fn parse_string_or_array(value: Value, field_name: &str) -> Result<Option<Vec<String>>> {
    match value {
        Value::Array(array) => {
            Some(array.iter()
                .map(|v| -> Result<String, anyhow::Error> { 
                    Ok(v.as_str()
                        .with_context(|| format!("{} array element must be a string", field_name))?
                        .to_string()) 
                })
                .collect::<Result<Vec<_>, _>>()?)
        }
        Value::String(single) => Some(vec![single]),
        _ => anyhow::bail!("{} must be a string or array of strings", field_name),
    }
}

Also applies to: 273-285


452-452: Consider adding validation for comma-separated inputs.

The CLI parsing splits on commas and trims, but doesn't validate that the resulting strings are non-empty or valid IPs/CIDRs.

- ip: cli.ip.map(|s| s.split(',').map(|s| s.trim().to_string()).collect()),
+ ip: cli.ip.map(|s| s.split(',')
+     .map(|s| s.trim())
+     .filter(|s| !s.is_empty())
+     .map(String::from)
+     .collect()),

Also applies to: 459-459

src/ocp_postprocess/ip_rename/etcd_rename.rs (1)

13-16: Consider using the standard library's IP parsing directly.

The helper could be simplified by leveraging the IP address parsing more directly.

-fn is_ipv6(ip: &str) -> bool {
-    ip.parse::<IpAddr>().map(|addr| addr.is_ipv6()).unwrap_or(false)
-}
+fn is_ipv6(ip: &str) -> bool {
+    ip.parse::<Ipv6Addr>().is_ok()
+}
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f15bfcd and c4a40a4.

📒 Files selected for processing (9)
  • .gitignore (0 hunks)
  • hack/dummy_config.yaml (2 hunks)
  • src/config.rs (4 hunks)
  • src/config/cli.rs (2 hunks)
  • src/etcd_encoding.rs (3 hunks)
  • src/ocp_postprocess.rs (3 hunks)
  • src/ocp_postprocess/ip_rename.rs (2 hunks)
  • src/ocp_postprocess/ip_rename/etcd_rename.rs (9 hunks)
  • src/ocp_postprocess/machine_config_cidr_rename/etcd_rename.rs (2 hunks)
💤 Files with no reviewable changes (1)
  • .gitignore
🔇 Additional comments (16)
src/config/cli.rs (1)

100-100: LGTM: Clear dual-stack documentation.

The documentation updates clearly explain the expected comma-separated format for dual-stack configurations and properly specify the IPv4-first ordering requirement.

Also applies to: 153-153

hack/dummy_config.yaml (1)

45-47: LGTM: Comprehensive dual-stack configuration example.

The dummy configuration properly demonstrates dual-stack networking with consistent IPv4-first ordering across all network components (IPs, cluster networks, machine networks, and service networks).

Also applies to: 74-82, 89-91

src/config.rs (1)

45-45: LGTM: Proper data structure changes for dual-stack support.

The field type changes from Option<String> to Option<Vec<String>> correctly support multiple IPs and CIDRs for dual-stack networking.

Also applies to: 53-53

src/ocp_postprocess.rs (3)

155-163: LGTM! Well-structured dual-stack IP handling.

The implementation correctly handles both single IP (backward compatible) and dual-stack scenarios with appropriate logging.


205-211: LGTM! Clean handling of multiple machine network CIDRs.

The approach of joining CIDRs with commas maintains backward compatibility while supporting dual-stack configurations.


924-936: LGTM! Consistent implementation pattern.

The new function follows the established pattern for rename operations with proper error context.

src/ocp_postprocess/ip_rename.rs (4)

18-30: Good separation of concerns with pre-extraction pattern.

The approach of extracting original IPs before making any modifications prevents potential race conditions and ensures data integrity.


86-140: Excellent refactoring for code reuse.

The helper function effectively consolidates all etcd fixes and enables both single and dual-stack code paths to share the same logic.


142-186: Clean separation of IP extraction logic.

The function properly extracts the original IP without side effects and correctly handles IPv6 bracket formatting.


198-220: Well-structured dual-stack processing with proper validation.

The function correctly validates the presence of both IPs and provides clear logging for troubleshooting.

src/ocp_postprocess/ip_rename/etcd_rename.rs (6)

26-66: Well-implemented dual-stack IP extraction logic.

The function correctly handles node address extraction with proper validation for single-node scenarios and clear error messages.


84-110: Excellent defensive programming with conditional replacement.

The function properly validates the presence of the original IP before making changes, preventing unintended modifications.


114-140: Robust URL replacement logic.

The function correctly validates URL presence before replacement and handles IPv6 formatting consistently.


486-490: Good handling of optional IPv6 arguments in dual-stack.

The function correctly handles the case where IPv6 arguments might not be present in certain configurations, with clear logging for operators.


538-580: Excellent refactoring of etcd member update logic.

The function now properly queries etcd state, validates single-node assumptions, and only updates when the original IP is actually present. This is much more robust than blind replacement.


27-27: Confirm deprecated “minions” prefix in etcd key lookup

  • File: src/ocp_postprocess/ip_rename/etcd_rename.rs, line 27
    let node_keys = etcd_client.list_keys("minions").await?;
  • The term "minions" was renamed to "nodes" in Kubernetes v1.0+.
  • Please verify whether your OpenShift/Kubernetes version still stores node entries under "minions" in etcd. If it now uses "nodes", update this call accordingly.

Copy link
Copy Markdown
Member

@omertuc omertuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, from skimming looks generally OK so far, didn't read too carefully. A few comments / questions

ip.parse::<IpAddr>().map(|addr| addr.is_ipv6()).unwrap_or(false)
}

// Extract both original IPv4 and IPv6 IPs from dual-stack cluster node configuration
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have this information provided to us, we already get the original IP from LCA (which gets it from the seed manifest seed image label), why do we need to extract it from the nodes?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we get it ? do you mean in the cn_san_replace_rules ? otherwise AFAIU the recert config only contains desired state not seed state no ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nvm, I thought we get it as an explicit argument, but we just extract it from the cluster. Ignore me

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 22, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This PR implements comprehensive dual stack networking support in the recert tool, enabling transformation from one dual stack configuration to another with conditional replacement logic.

Overview

The recert tool now supports dual stack networking configurations, allowing clusters to operate with both IPv4 and IPv6 addresses simultaneously. This includes intelligent conditional replacement that only modifies resources when the original IP addresses are actually present.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Conditional Replacement Logic (Major Enhancement)

Problem Solved

Previously, etcd resource modification functions would unconditionally replace IP addresses, potentially corrupting configurations when original IPs were not present.

Solution Implemented

  • Conditional replacement: All etcd modification functions now check if the original IP exists before performing replacements
  • Fail-safe behavior: Functions skip replacement and log informational messages when original IPs are not found
  • No failures on mismatches: Missing original IPs no longer cause the process to fail

Functions Updated with Conditional Logic

  • fix_etcd_endpoints: Only replaces endpoint IPs when original IP matches current value
  • fix_openshift_apiserver_configmap: Checks storage config URLs before modification
  • fix_openshiftapiservers_cluster: Validates storage config contains original IP
  • fix_networks_cluster: Verifies annotation values before updating
  • fix_etcd_member: Fetches actual member configuration and validates peer URLs
  • fix_storage_config: Helper function now requires original IP verification

Technical Implementation

// Example: Conditional replacement pattern
if current_value == original_ip {
   // Perform replacement
   data[key] = new_ip;
   put_etcd_yaml(etcd_client, &location, resource).await?;
} else {
   log::info!("Original IP {} not found, current value is {}, skipping replacement", 
              original_ip, current_value);
}

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Dual Stack IPs:
  • First IP (IPv4) processed through all etcd resources
  • Second IP (IPv6) processed through all etcd resources
  • Each IP change is logged for traceability

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Etcd Resource Management

  1. Original IP extraction: Retrieves current IPs from node configuration before making changes
  2. Sequential processing: IPv4 and IPv6 changes applied in separate passes
  3. Resource isolation: Each IP family processed independently to prevent cross-contamination

Function Reorganization

  • Centralized etcd modifications: All etcd resource changes moved to fix_etcd_resources_for_ip_pair
  • Consistent API: All functions now take both original_ip and new_ip parameters
  • Improved maintainability: Common patterns extracted and standardized

Backward Compatibility

  • Single IP support: Existing single IP configurations continue to work unchanged
  • CLI compatibility: Single IP values work as before, arrays enable dual stack
  • Configuration files: Both single values and arrays supported in YAML

Benefits

  1. Robust dual stack support: Full IPv4/IPv6 dual stack cluster transformations
  2. Safe operations: Conditional replacement prevents accidental corruption
  3. Better observability: Detailed logging shows what changes are made or skipped
  4. Maintainable code: Consistent patterns across all etcd modification functions
  5. Future-proof: Architecture supports easy addition of new dual stack features

Testing

  • All existing single IP tests continue to pass
  • New dual stack scenarios covered by expanded test cases
  • Conditional replacement logic prevents regressions from missing original IPs

Summary by CodeRabbit

  • New Features

  • Added support for dual-stack (IPv4 and IPv6) cluster configuration, including handling multiple IP addresses and network CIDRs in both configuration files and command-line inputs.

  • Introduced comprehensive dual-stack IP renaming functionality, ensuring both IPv4 and IPv6 addresses are processed in etcd and filesystem resources.

  • Enhanced etcd encoding/decoding to support additional Kubernetes resource types.

  • Documentation

  • Updated command-line option descriptions to clarify input formats for dual-stack IPs and CIDRs.

  • Bug Fixes

  • Improved logic to ensure IP and CIDR replacements only occur when the original values are present, with robust handling for both IPv4 and IPv6 scenarios.

  • Refactor

  • Streamlined internal logic for processing multiple network entries and improved logging for dual-stack operations.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
src/ocp_postprocess/ip_rename/etcd_rename.rs (1)

17-21: Address the past review comment about IP extraction.

There's a valid concern from a previous review questioning why we need to extract original IPs from nodes when this information should be provided by LCA. This adds complexity and potential points of failure.

Consider whether this extraction logic is necessary or if the original IP information can be obtained from LCA as suggested in the previous review.

🧹 Nitpick comments (4)
src/ocp_postprocess/ip_rename/etcd_rename.rs (3)

12-15: Consider more robust IPv6 detection.

The current implementation will return false for invalid IP strings, which might mask parsing errors. Consider being more explicit about error handling.

-fn is_ipv6(ip: &str) -> bool {
-    ip.parse::<IpAddr>().map(|addr| addr.is_ipv6()).unwrap_or(false)
-}
+fn is_ipv6(ip: &str) -> Result<bool> {
+    let addr = ip.parse::<IpAddr>().context("Invalid IP address format")?;
+    Ok(addr.is_ipv6())
+}

Alternatively, if you want to keep the current signature for simplicity, add a comment explaining the behavior for invalid IPs.


23-78: The IP extraction logic is well-implemented but has room for improvement.

The function correctly handles both single-stack and dual-stack scenarios with proper error handling and logging.

However, consider these improvements:

  1. The function assumes IPv4 comes before IPv6, but this might not always be guaranteed in node addresses:
 let mut result = Vec::new();
 
-if let Some(ipv4) = original_ipv4 {
-    result.push(ipv4);
-}
-
-if let Some(ipv6) = original_ipv6 {
-    result.push(ipv6);
-}
+// Ensure consistent ordering: IPv4 first, then IPv6
+if let Some(ipv4) = original_ipv4 {
+    result.push(ipv4);
+}
+if let Some(ipv6) = original_ipv6 {
+    result.push(ipv6);
+}
  1. Consider adding validation for the extracted IP addresses to ensure they're valid.

550-591: Excellent rewrite using etcd client API with robust IPv6 support.

The complete rewrite to use the etcd client API instead of direct manipulation is a significant improvement. The function properly:

  • Validates there's exactly one member (correct for SNO)
  • Uses proper IPv6 address parsing instead of string manipulation
  • Only updates when the original IP is found in peer URLs
  • Provides detailed logging

Minor suggestion for consistency:

-    let contains_original = current_member.peer_urls().iter().any(|url| url == &expected_original_url);
+    let contains_original = current_member.peer_urls().iter().any(|url| *url == expected_original_url);

This avoids the unnecessary reference in the comparison.

src/ocp_postprocess/machine_config_cidr_rename/etcd_rename.rs (1)

34-42: Good implementation of dual-stack CIDR support.

The changes correctly enable dual-stack support by splitting comma-separated CIDRs and replacing the machine network array. The approach is clean and handles both single-stack and dual-stack scenarios.

Consider adding validation for the CIDR format:

 let new_networks: Vec<_> = machine_config_network
     .split(',')
-    .map(|cidr| serde_json::json!({"cidr": cidr.trim()}))
+    .map(|cidr| {
+        let trimmed = cidr.trim();
+        // Basic CIDR validation could be added here if needed
+        serde_json::json!({"cidr": trimmed})
+    })
     .collect();

This would provide a place to add CIDR validation if needed in the future.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c4a40a4 and ce0c1d7.

📒 Files selected for processing (9)
  • .gitignore (0 hunks)
  • hack/dummy_config.yaml (2 hunks)
  • src/config.rs (4 hunks)
  • src/config/cli.rs (2 hunks)
  • src/etcd_encoding.rs (3 hunks)
  • src/ocp_postprocess.rs (3 hunks)
  • src/ocp_postprocess/ip_rename.rs (3 hunks)
  • src/ocp_postprocess/ip_rename/etcd_rename.rs (9 hunks)
  • src/ocp_postprocess/machine_config_cidr_rename/etcd_rename.rs (2 hunks)
💤 Files with no reviewable changes (1)
  • .gitignore
✅ Files skipped from review due to trivial changes (1)
  • src/config/cli.rs
🚧 Files skipped from review as they are similar to previous changes (5)
  • src/config.rs
  • src/etcd_encoding.rs
  • hack/dummy_config.yaml
  • src/ocp_postprocess/ip_rename.rs
  • src/ocp_postprocess.rs
🔇 Additional comments (6)
src/ocp_postprocess/ip_rename/etcd_rename.rs (6)

80-124: Excellent improvement with conditional replacement logic.

The addition of conditional replacement logic significantly improves safety by only modifying configurations when the original IP is actually present. The IPv6 bracket formatting is handled correctly.

This prevents potential configuration corruption that could occur with unconditional replacements when the original IP isn't found in the expected location.


126-152: Well-implemented conditional replacement with proper IPv6 handling.

The function correctly handles IPv6 address formatting with brackets and only performs replacements when the original IP is found in the current configuration.

The logging provides good visibility into when replacements are skipped, which will be helpful for debugging.


204-255: Consistent implementation of conditional replacement logic.

The function properly implements the same conditional replacement pattern as other functions, only modifying configurations when the original IP is found.

The logging provides good visibility into when replacements are skipped.


459-518: Good handling of dual-stack scenarios with informative logging.

The function correctly handles cases where the etcd server argument might not be found (normal for IPv6 in dual-stack scenarios) with appropriate logging. The IPv6 formatting with brackets and quotes is handled properly.

The early return when the argument is not found prevents unnecessary processing and provides clear feedback about what's happening.


427-440: Clean integration with updated storage config logic.

The function correctly delegates to the updated fix_storage_config function that now includes conditional replacement logic.


520-548: Proper conditional replacement for network cluster annotations.

The function correctly implements conditional replacement logic, only updating the annotation when the original IP is found. The logging provides good visibility into different scenarios (annotation not found vs. value mismatch).

@danmanor
Copy link
Copy Markdown
Member Author

@omertuc Can you please add /ok-to-test ?

@danmanor danmanor requested a review from omertuc July 22, 2025 12:37
@mresvanis
Copy link
Copy Markdown

/ok-to-test

danmanor added a commit to danmanor/lifecycle-agent that referenced this pull request Jul 22, 2025
Background / Context

The lifecycle-agent is a core component responsible for managing Image-Based Upgrades (IBU) and Image-Based Install (IBI) operations in OpenShift Single Node OpenShift (SNO) clusters. It handles critical cluster lifecycle operations including:

- **Network Configuration Management**: Configuring node IPs, machine networks, cluster networks, and service networks during cluster transitions
- **Recertification (Recert)**: Re-signing certificates with updated cluster information (IPs, hostnames, etc.) when transforming seed images into target clusters
- **Post-Pivot Operations**: Managing network setup, DNS configuration, and kubelet configuration after cluster pivot operations
- **Seed Cluster Information**: Capturing and transforming network details from seed clusters for target cluster deployment

Currently, the lifecycle-agent's network handling is designed around single-stack networking, where clusters operate on either IPv4 OR IPv6, but not both simultaneously. All network-related data structures use single string fields (`NodeIP`, `MachineNetwork`) and the logic assumes a single IP address per network interface.

Key components involved:
- `api/seedreconfig`: Defines the `SeedReconfiguration` API for cluster transformation parameters
- `utils/client_helper.go`: Extracts cluster network information from Kubernetes API
- `lca-cli/postpivot`: Handles post-upgrade network configuration and kubelet setup
- `internal/recert`: Manages certificate re-signing with updated network information
- `lca-cli/seedclusterinfo`: Captures seed cluster network details for replication

Issue / Requirement / Reason for change

**[MGMT-21201](https://issues.redhat.com//browse/MGMT-21201)**: The lifecycle-agent needs to support dual-stack networking configurations where OpenShift clusters operate with both IPv4 and IPv6 addresses simultaneously on the same interfaces.

Current Limitations:
1. **Single IP Assumption**: All network fields (`NodeIP`, `MachineNetwork`) are single strings, preventing multiple IP support
2. **Limited Network Discovery**: Cluster info extraction only captures the first internal IP address
3. **Inadequate Recert Logic**: Certificate re-signing only handles single IP changes
4. **Kubelet Configuration**: Node IP hint generation assumes single network per stack
5. **Missing Test Coverage**: No validation for dual-stack scenarios

Requirements:
- Support IPv4 + IPv6 dual-stack clusters in IBU/IBI operations
- Maintain 100% backward compatibility with existing single-stack configurations
- Handle multiple machine networks per IP family
- Update recert logic to process multiple IP addresses in certificate SANs
- Ensure proper kubelet configuration for dual-stack node IPs

Changes Made

API Extensions
- **Added `NodeIPs []string`** to `SeedReconfiguration` and `SeedClusterInfo` for multiple node IPs
- **Added `MachineNetworks []string`** to support multiple machine network CIDRs
- **Preserved legacy fields** (`NodeIP`, `MachineNetwork`) for backward compatibility with precedence rules

Network Configuration Updates
- **Enhanced `GetClusterInfo()`** to discover all internal node IPs via `getNodeInternalIPs()`
- **Added `getMachineNetworks()`** to extract all machine networks from install config
- **Implemented backward compatibility** by populating legacy fields with first array element

Post-Pivot Improvements
- **Updated `setNodeIpHint()`** to generate space-separated IP hints: `KUBELET_NODEIP_HINT=<ip1> <ip2>`
- **Refactored `setNodeIPIfNotProvided()`** to parse kubelet config from `/etc/systemd/system/kubelet.service.d/20-nodenet.conf`
- **Added `parseKubeletNodeIPs()`** function to extract both `KUBELET_NODE_IP` and `KUBELET_NODE_IPS` environment variables
- **Enhanced file validation** to check for both existence AND valid content before triggering `nodeip-configuration` service

Recertification Logic
- **Implemented `slices.Equal()`** comparison for clean IP change detection
- **Updated config IP format** to comma-separated list: `config.IP = "ip1,ip2"` for multiple addresses
- **Enhanced certificate SAN rules** to include all old and new IP addresses in replacement rules

Test Scenarios Covered:
- ✅ Single-stack IPv4 and IPv6 configurations
- ✅ Dual-stack (IPv4 + IPv6) with both primary orders
- ✅ Multiple networks per IP family
- ✅ Legacy → new field migration scenarios
- ✅ Error handling and edge cases
- ✅ Backward compatibility validation

Backward Compatibility

**100% backward compatibility maintained**:
- Existing single-stack clusters continue to work without modification
- Legacy `NodeIP` and `MachineNetwork` fields preserved and populated
- New fields (`NodeIPs`, `MachineNetworks`) take precedence when specified
- All existing APIs and behavior unchanged for single-stack scenarios

Testing

- **All existing tests pass** - no regressions introduced
- **59 new test cases** covering all dual-stack scenarios
- **Production-ready validation** for IPv4, IPv6, and dual-stack configurations
- **Edge case coverage** including empty configs, invalid data, and service interactions

Other related PRs

recert - rh-ecosystem-edge/recert#390
Copy link
Copy Markdown
Member

@omertuc omertuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, a few more small things I noticed but otherwise lgtm

@danmanor danmanor force-pushed the enable-dual-stack branch from e9c011b to 15438a9 Compare July 31, 2025 07:33
@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 31, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This PR implements comprehensive dual stack networking support in the recert tool, enabling transformation from one dual stack configuration to another with conditional replacement logic.

Overview

The recert tool now supports dual stack networking configurations, allowing clusters to operate with both IPv4 and IPv6 addresses simultaneously. This includes intelligent conditional replacement that only modifies resources when the original IP addresses are actually present.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Conditional Replacement Logic (Major Enhancement)

Problem Solved

Previously, etcd resource modification functions would unconditionally replace IP addresses, potentially corrupting configurations when original IPs were not present.

Solution Implemented

  • Conditional replacement: All etcd modification functions now check if the original IP exists before performing replacements
  • Fail-safe behavior: Functions skip replacement and log informational messages when original IPs are not found
  • No failures on mismatches: Missing original IPs no longer cause the process to fail

Functions Updated with Conditional Logic

  • fix_etcd_endpoints: Only replaces endpoint IPs when original IP matches current value
  • fix_openshift_apiserver_configmap: Checks storage config URLs before modification
  • fix_openshiftapiservers_cluster: Validates storage config contains original IP
  • fix_networks_cluster: Verifies annotation values before updating
  • fix_etcd_member: Fetches actual member configuration and validates peer URLs
  • fix_storage_config: Helper function now requires original IP verification

Technical Implementation

// Example: Conditional replacement pattern
if current_value == original_ip {
   // Perform replacement
   data[key] = new_ip;
   put_etcd_yaml(etcd_client, &location, resource).await?;
} else {
   log::info!("Original IP {} not found, current value is {}, skipping replacement", 
              original_ip, current_value);
}

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Dual Stack IPs:
  • First IP (IPv4) processed through all etcd resources
  • Second IP (IPv6) processed through all etcd resources
  • Each IP change is logged for traceability

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Etcd Resource Management

  1. Original IP extraction: Retrieves current IPs from node configuration before making changes
  2. Sequential processing: IPv4 and IPv6 changes applied in separate passes
  3. Resource isolation: Each IP family processed independently to prevent cross-contamination

Function Reorganization

  • Centralized etcd modifications: All etcd resource changes moved to fix_etcd_resources_for_ip_pair
  • Consistent API: All functions now take both original_ip and new_ip parameters
  • Improved maintainability: Common patterns extracted and standardized

Backward Compatibility

  • Single IP support: Existing single IP configurations continue to work unchanged
  • CLI compatibility: Single IP values work as before, arrays enable dual stack
  • Configuration files: Both single values and arrays supported in YAML

Benefits

  1. Robust dual stack support: Full IPv4/IPv6 dual stack cluster transformations
  2. Safe operations: Conditional replacement prevents accidental corruption
  3. Better observability: Detailed logging shows what changes are made or skipped
  4. Maintainable code: Consistent patterns across all etcd modification functions
  5. Future-proof: Architecture supports easy addition of new dual stack features

Testing

  • All existing single IP tests continue to pass
  • New dual stack scenarios covered by expanded test cases
  • Conditional replacement logic prevents regressions from missing original IPs

Summary by CodeRabbit

  • New Features

  • Added full support for dual-stack (IPv4 and IPv6) network configurations, including handling of multiple IP addresses and network CIDRs.

  • Introduced dual-stack IP renaming capabilities for both etcd resources and filesystem resources.

  • Added support for Kubernetes Node resource handling in etcd encoding/decoding.

  • Improvements

  • Enhanced logging to clarify which IPs and CIDRs are being processed.

  • Improved robustness in IP and CIDR processing, ensuring changes only occur when expected values are present.

  • Updated documentation to guide users on providing multiple IPs/CIDRs for dual-stack setups.

  • Refined error handling using consistent assertions for validation checks.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@danmanor danmanor requested a review from omertuc July 31, 2025 07:42
@danmanor
Copy link
Copy Markdown
Member Author

/retest

Copy link
Copy Markdown
Member

@omertuc omertuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One final thing

@danmanor danmanor force-pushed the enable-dual-stack branch 2 times, most recently from 7d0cbef to 368516c Compare July 31, 2025 08:51
@danmanor danmanor requested a review from omertuc July 31, 2025 08:52
@omertuc
Copy link
Copy Markdown
Member

omertuc commented Jul 31, 2025

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Jul 31, 2025
@omertuc
Copy link
Copy Markdown
Member

omertuc commented Jul 31, 2025

/approve

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Jul 31, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danmanor, omertuc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot
Copy link
Copy Markdown
Collaborator

openshift-ci-robot commented Jul 31, 2025

@danmanor: This pull request references MGMT-21200 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Dual Stack Networking Support for Recert

This PR implements comprehensive dual stack networking support in the recert tool, enabling transformation from one dual stack configuration to another with conditional replacement logic.

Overview

The recert tool now supports dual stack networking configurations, allowing clusters to operate with both IPv4 and IPv6 addresses simultaneously. This includes intelligent conditional replacement that only modifies resources when the original IP addresses are actually present.

Key Components Supporting Dual Stack

  1. API VIPs: Both IPv4 and IPv6 virtual IP addresses for API server access
  2. Ingress VIPs: Both IPv4 and IPv6 virtual IP addresses for ingress/router traffic
  3. Machine Networks: Multiple network CIDR blocks for different IP families
  4. Cluster Networks: Multiple overlay network CIDR blocks for pods
  5. Service Networks: Multiple CIDR blocks for Kubernetes services

Changes Made

1. Configuration Structure Changes

  • Updated ClusterCustomizations struct: Changed ip field from Option<String> to Option<Vec<String>>
  • Updated machine_network_cidr field: Changed from Option<String> to Option<Vec<String>>
  • Enhanced parsing logic: Now supports both single values (backward compatibility) and arrays for dual stack
  • CLI support: Comma-separated values via --ip 192.168.1.100,2001:db8::100 and --machine-network-cidr 192.168.1.0/24,2001:db8::/64

2. Network Processing Updates

  • Removed dual stack restrictions: Eliminated the error that prevented processing dual stack clusters
  • Enhanced machine network handling: Now supports multiple network entries in install-config
  • IP transformation logic: Each IP in the array is processed sequentially

3. Conditional Replacement Logic (Major Enhancement)

Problem Solved

Previously, etcd resource modification functions would unconditionally replace IP addresses, potentially corrupting configurations when original IPs were not present.

Solution Implemented

  • Conditional replacement: All etcd modification functions now check if the original IP exists before performing replacements
  • Fail-safe behavior: Functions skip replacement and log informational messages when original IPs are not found
  • No failures on mismatches: Missing original IPs no longer cause the process to fail

Functions Updated with Conditional Logic

  • fix_etcd_endpoints: Only replaces endpoint IPs when original IP matches current value
  • fix_openshift_apiserver_configmap: Checks storage config URLs before modification
  • fix_openshiftapiservers_cluster: Validates storage config contains original IP
  • fix_networks_cluster: Verifies annotation values before updating
  • fix_etcd_member: Fetches actual member configuration and validates peer URLs
  • fix_storage_config: Helper function now requires original IP verification

Technical Implementation

// Example: Conditional replacement pattern
if current_value == original_ip {
   // Perform replacement
   data[key] = new_ip;
   put_etcd_yaml(etcd_client, &location, resource).await?;
} else {
   log::info!("Original IP {} not found, current value is {}, skipping replacement", 
              original_ip, current_value);
}

Processing Logic

IP Address Processing

  1. Single IP: Processed once, maintains backward compatibility
  2. Dual Stack IPs:
  • First IP (IPv4) processed through all etcd resources
  • Second IP (IPv6) processed through all etcd resources
  • Each IP change is logged for traceability

Network Configuration Processing

  1. Machine networks: Supports comma-separated CIDRs in machine_network_cidr parameter
  2. Install config: Automatically handles arrays of network configurations
  3. Validation: Removed restrictions that blocked dual stack processing

Etcd Resource Management

  1. Original IP extraction: Retrieves current IPs from node configuration before making changes
  2. Sequential processing: IPv4 and IPv6 changes applied in separate passes
  3. Resource isolation: Each IP family processed independently to prevent cross-contamination

Function Reorganization

  • Centralized etcd modifications: All etcd resource changes moved to fix_etcd_resources_for_ip_pair
  • Consistent API: All functions now take both original_ip and new_ip parameters
  • Improved maintainability: Common patterns extracted and standardized

Backward Compatibility

  • Single IP support: Existing single IP configurations continue to work unchanged
  • CLI compatibility: Single IP values work as before, arrays enable dual stack
  • Configuration files: Both single values and arrays supported in YAML

Benefits

  1. Robust dual stack support: Full IPv4/IPv6 dual stack cluster transformations
  2. Safe operations: Conditional replacement prevents accidental corruption
  3. Better observability: Detailed logging shows what changes are made or skipped
  4. Maintainable code: Consistent patterns across all etcd modification functions
  5. Future-proof: Architecture supports easy addition of new dual stack features

Testing

  • All existing single IP tests continue to pass
  • New dual stack scenarios covered by expanded test cases
  • Conditional replacement logic prevents regressions from missing original IPs

Summary by CodeRabbit

  • New Features

  • Added support for dual-stack (IPv4 and IPv6) network configurations, including handling multiple IP addresses and network CIDRs throughout the configuration and processing logic.

  • Enhanced IP renaming and network CIDR update processes to work with both IPv4 and IPv6 addresses for improved cluster compatibility.

  • Bug Fixes

  • Improved robustness of IP and CIDR replacement logic to ensure changes only occur when the original values are present.

  • Documentation

  • Updated configuration documentation to clarify dual-stack usage and ordering requirements for IP addresses and CIDRs.

  • Style

  • Standardized error handling for better consistency and maintainability.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@omertuc
Copy link
Copy Markdown
Member

omertuc commented Jul 31, 2025

/ok-to-test

@danmanor danmanor force-pushed the enable-dual-stack branch from 368516c to 3ee6a5a Compare July 31, 2025 11:47
@openshift-ci openshift-ci bot removed the lgtm label Jul 31, 2025
@omertuc
Copy link
Copy Markdown
Member

omertuc commented Jul 31, 2025

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Jul 31, 2025
@danmanor
Copy link
Copy Markdown
Member Author

/override ci/prow/e2e-aws-ovn-single-node-recert-parallel

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Jul 31, 2025

@danmanor: danmanor unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:.

Details

In response to this:

/override ci/prow/e2e-aws-ovn-single-node-recert-parallel

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@mresvanis
Copy link
Copy Markdown

/override ci/prow/e2e-aws-ovn-single-node-recert-parallel ci/prow/e2e-aws-ovn-single-node-recert-serial

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Jul 31, 2025

@mresvanis: Overrode contexts on behalf of mresvanis: ci/prow/e2e-aws-ovn-single-node-recert-parallel, ci/prow/e2e-aws-ovn-single-node-recert-serial

Details

In response to this:

/override ci/prow/e2e-aws-ovn-single-node-recert-parallel ci/prow/e2e-aws-ovn-single-node-recert-serial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-merge-bot openshift-merge-bot bot merged commit f409769 into rh-ecosystem-edge:main Jul 31, 2025
13 of 14 checks passed
@coderabbitai coderabbitai bot mentioned this pull request Dec 22, 2025
@coderabbitai coderabbitai bot mentioned this pull request Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants